Computer implemented method for guiding traffic participants

ABSTRACT

A computer implemented method for guiding traffic participants, especially pedestrians, especially visually impaired and blind people, especially for guiding in urban environments, between at least two places wherein the method contains the following steps: A. Providing a multi-modal three-dimensional map; B. Calculating a route based on the multi-modal three-dimensional map connecting the at least two places over at least one intermediate waypoint; C. Determining precise location of the traffic participant, preferably by using the multi-modal three-dimensional map; and D. Setting beacons along the path at the waypoints.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to a computer implemented method for guiding atraffic participant, especially a pedestrian, especially a visuallyimpaired or blind person, especially for guiding in urban environments,between at least two places.

Description of the Related Art

With the public accessibility of the Global Positioning System (GPS) andthe development of continuously stronger, transportable computationaldevices—such as smartphones—the use of computerized navigational systemshas become wide spread among motorized and unmotorized trafficparticipants alike. However, visually impaired and blind people whoprobably need assistance the most when navigating in traffic cannot usethe current navigational systems.

SUMMARY OF THE INVENTION

Hence it is an object of the invention to overcome the aforementionedobstacles and drawbacks and to provide a navigational system that can beused by any traffic participants and in particular by visually impairedand blind people.

This problem is solved by the method disclosed and claimed.

Further preferred and advantageous embodiments of the invention are alsodisclosed.

In the following a depiction of a preferred embodiment of the inventionand the problems within the current technology are described.

The described embodiments and aspects are solely meant to exemplify theinvention and its related problems, which is not limited to the shownexamples but may be implemented in a wide range of applications.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the included drawings,in which:

FIG. 1 shows a draft urban scene exemplifying the problem,

FIG. 2 shows the scene from FIG. 1 with a path that is generatedaccording to the invention,

FIG. 3 another urban scene with a path that is generated according tothe invention,

FIG. 4 a two-dimensional map,

FIG. 5 an aerial view the region shown in FIG. 4,

FIG. 6 a conventional three-dimensional map of the region shown in FIG.4,

FIG. 7 a depth image of the approximate region shown in FIG. 4,

FIG. 8 a multi-modal three-dimensional map of a section of the regionshown in FIG. 4,

FIG. 9 the multi-modal three-dimensional map of FIG. 8 with a path thatis generated according to the invention and

FIG. 10 a flow chart roughly depicting a method of localizing a user ofthe system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 shows an urban scene 1 with a pavement 2, a building 3 that isbuilt closer to the street, a building 4 that is built more distant fromthe street, a flower bed 5, lamp posts 6 and parked cars 7. FIG. 1 alsoshows an exemplary route 8 as it might be calculated by a conventionalnavigational system. It can be clearly seen, that the actual route takenby a pedestrian normally would deviate from the shown straight line andtake the flower bed 5 and the building 3 closer to the street intoconsideration. However, this action requires the pedestrian to knowabout the flower bed 5 or the irregularly placed houses 3, 4. Thisinformation can be easily acquired by simply being at the scene andseeing the placement of the aforementioned urban elements. But with thelack of visual information to assess the scene the given route 8 isinsufficient and can even be dangerous, for example by causing a blindperson to stumble over the edge of the flower bed 5 which occasionallycannot even be detected by a white cane.

The scene shown in FIG. 2 is the same as shown in FIG. 1 with animproved path 9, that—if followed by a pedestrian—requires no adjustmentdue to obstacles. The flower bed 5 and the differently placed buildings3, 4 are no longer a danger.

Two other problems are illustrated in FIG. 3. Firstly, there is theproblem that arises when a visually impaired pedestrian wants to cross astreet 10. Given that safety is of paramount importance, any route thatrequires the crossing of streets must be built so that zebra crossings11 are always prioritized over other routes. Secondly, especially inurban surroundings, stationary installations like bollards 12, lampposts 6 (see FIG. 1), trash bins 13, street signs 14 and the like aresafety hazards that have to be circumvented. An optimized path 15 thatmeets these requirements is also shown in FIG. 3.

To create and efficiently use such an optimized path 9, 15 three majorproperties of the system are highly advantageous. Firstly, a route asprecise and fine-granular as possible is to be defined on a map.Secondly, the location of the user ought to be known with very highaccuracy and within very fine measurements, approximately within only afew centimeters, in order for a user to be able to follow the route. Theusual “couple of meters”—accuracy provided by current technology such asGPS or magnetic sensors (mobile device compasses) are not sufficient.Thirdly, in order for a blind or visually impaired person to make use ofthe path information, it is to be conveyed in a way such that such aperson can use it.

To accomplish this task, a guiding system for visually impaired or blindpeople according to the invention must include carrying out the Steps:

-   -   A. providing a multi-modal three-dimensional map,    -   B. calculating a route based on the multi-modal        three-dimensional map connecting the at least two places over at        least one intermediate waypoint,    -   C. determining a precise location of the traffic participant,        preferably by using the multi-modal three-dimensional map and    -   D. setting beacons along the path at the waypoints.

A multi-modal three-dimensional map as it is provided in step A isderived from two or more different sources whereby each source addsanother layer of information. A precise but conventional map may containinformation on where the pavements and streets are, but it probably doesnot discern between pavement and flowerbeds. A (municipal) treeinventory can provide information on the location of trees; this isoften combined with a garden and parks department where the preciselocation of public flower beds is charted. Municipal utilities canprovide information on electricity (street lamps) and water (hydrants,manholes). The department responsible for traffic can provide planswhere traffic lights, zebra crossings and the like are located.

Next to the aforementioned cartographic and geodetic information othersources can be added to a multi-modal three-dimensional map, such as:areal views, satellite pictures, surface texture data or conventional3D-data. The latter being plain geometric information onthree-dimensional objects like for example the shape and location ofhouses.

The list of possible sources to create a multi-modal three-dimensionalmap is not exhaustive and can be expanded to reflect the distinctivepeculiarities of a city or region, for example cycling tracks on thepavements, spaces reserved for horse carriages, tramway tracks, stairs,the type of paving (especially cobble stones), monuments, (park)benches, drinking fountains, trash bins, bicycle stands, outdoor diningareas of restaurants or defibrillators.

FIG. 4 to FIG. 6 show different examples for freely available maps andinformation that can be combined into a multi-modal three-dimensionalmap. FIG. 4 shows a two-dimensional map. FIG. 5 shows a reconstructedareal view, that is derived from satellite pictures with many differentdatasets from different satellites with different sensors. FIG. 6 showsa three-dimensional model of the region depicted in FIG. 4 and FIG. 5.FIG. 7 shows a so called “depth image” of the region shown in FIG. 4 toFIG. 6. A depth image shows distances along a defined axis, with moredistant surfaces usually being depicted darker and closer surfaces beingdepicted lighter. In the example shown the defined axis is the vertical.Brighter parts of the depth image are in conclusion higher and darkerparts flatter or lower. In the case of the urban scene the depth imagein FIG. 7 hence shows the height of buildings and other structures.

All the data layers can be obtained from a multitude of availablesources. A preferred way of obtaining data is through open sources. Forexample, the two-dimensional map data, the aerial view/map andconventional three-dimensional map data are available, in many caseswithout any usage limitation, e.g. from the OpenStreetMap Foundation.Moreover, other sources such as government institutions have publiclyavailable geographic data about cities.

For example, the city of Vienna has the three previously mentionedsources as well as the aforementioned surface model from which one canextract the height of objects present at each of the points in the map.In total the city of Vienna has over 50 different datasets with severallevels of detail that can be used to create a multi-modalthree-dimensional map.

In a preferred embodiment of the invention the different layers of themulti-modal three-dimensional map are combined in a spatially coherentway. Spatial coherence can for example be achieved by defining objectsand features, like houses, in a common coordinate system. By taking thisstep alone the layers are already aligned to some extent. Moreover, amore precise alignment can be obtained by using image registrationmethods, based on features present in both map layers (for examplebuildings in aerial view and two-dimensional layer) which are well knownin the art. With this alignment, the location of objects which are onlypresent in some layers (for example road limits or fire hydrants) can becorrelated with all the other map features in the multi-modal map.

If combined the following information can be extracted from thedifferent exemplary types of datasets shown in FIG. 4 to FIG. 7:

The three-dimensional layer, for example, can yield the information onthe precise location of buildings' walls 19 (see FIG. 8) or edges 20(see FIG. 8). A border 22 (see FIG. 9) between pavement 2 and street 10can be extracted from the aerial view. A combination between the depthimage and three-dimensional map yields very good estimations on thecorrect location of trees (see obstacles 21 in FIG. 8) for example. Allsources of information mentioned in this example that is located in thecity of Vienna are freely available open data from the city governmentand other non-governmental sources. A multi-modal three-dimensional mapcan also be very useful independently of the invention, for example whenautomatically guiding autonomous vehicles or planning the transport oflarge objects on the street.

According to a further embodiment of the invention, at least onewalkable space is defined within the multi-modal three-dimensional map.The walkable space can be according to a very simple example everypavement minus everything that is not pavement, for example pavementminus every bench, trash bin, sing post, lamp post, bollard, flower bed,etc. In this case the walkable spaces can be defined automatically. Ofcourse, the walkable spaces can also be defined manually.

Next to the essentially stationary, aforementioned objects other aspectscan also be taken into consideration. For example, the entrance area ofa very busy shop can be excluded from the walkable space andcircumvented.

Accordingly step B is preferably carried out based on the multi-modalthree-dimensional map especially based on the walkable space.

Also, when carrying out step B (calculating a path) two walkable spacescan be connected via at least one waypoint. Furthermore, if two or morewalkable spaces are not bordering each other within the multi-modalthree-dimensional map, at least one transitioning space is defined and atransitioning space bridges a gap between said two or more walkablespaces.

This is illustrated in FIG. 3. The pavement 2 is not directly borderingthe other remote pavement 16. They are connected via two waypoints 17,18. Between them there is a zebra crossing 11. In this case the zebracrossing 11 is the transitioning space that bridges the gap between thetwo pavements 2, 16. Other types of transitioning spaces can for examplebe traffic lights and way between them. However, even stairs or spaceswith difficult pavement like cobble stones can be transitioning spaces,if they are not considered to be a safe walking space.

Another aspect of the invention is illustrated in FIG. 8 and FIG. 9.They show a very simplified example of a two-dimensional view of amulti-modal three-dimensional map.

The multi-modal three-dimensional map shows the building 3, the street10 and the pavement 2 just as a normal map would show. However, grace tothe additional layers of information the correct location of thebuilding's 3 walls 19 and edges 20 are correctly noted with their actuallocation. Trees 21 have been registered and placed accordingly. The sameapplies for the now correctly noted border 22 between pavement 2 andstreet 10.

According to one preferred embodiment of the invention at least oneobstacle is identified and marked in the multimodal three-dimensionalmap and at least one waypoint is set to circumvent said obstacle.

As can be seen in FIG. 9 a path 15 has been created along severalwaypoints 23 that are circumventing obstacles, such as walls 19, corners20, trees 21, and avoiding the border 23 between street 10 and pavement2.

When setting the waypoints 23 automatically or manually it is importantto try to be at a maximum distance to any spaces that are not walkablespaces. An easy way to find suitable places for waypoints could be toidentify any bottleneck and to place the waypoints essentially in themiddle of the bottleneck to achieve a maximum distance to all spacesthat are not walkable spaces.

In order to follow the now created path, the precise location of theperson (or vehicle/drone) has to be known. This can preferably be doneby locating a device that is used to carry out the computer implementedmethod, for example a smartphone. However, the methods to locate devicesare not precise enough to safely tell where along a path the device islocated or if on or near the path at all.

One possible method to determine the precise location of the trafficparticipant during step C of the invention is shown in the flow-chart inFIG. 10.

The depicted preferred embodiment includes the following sub-steps:

-   -   a. acquiring a real view by processing at least one digital        picture of the surrounding,    -   b. generating at least one possible artificial view based on a        raw location, the raw location providing a scene that is part of        the multi-modal three-dimensional map and that is being depicted        to provide the artificial view,    -   c. comparing the artificial view and the real view,    -   d. if the artificial view and the real view are essentially the        same, providing location as the point of view the artificial        view was generated from, and completing the subprocess,    -   e. if the artificial view and the real view are not essentially        the same repeating the subprocess with at least one different        artificial view.

Step a. can be simply carried out by photographing the scene in front ofthe device that is used to carry out the process. If the device is asmartphone, the smartphone camera can be used.

Filming a scene is also considered taking photographs since filmingbasically is taking photographs with a (higher) frame rate.

A succession of pictures that is strung together to create a lager realview is also considered to be within the scope of the invention.

Then at least one artificial view is created. This can be done forexample by using a ray-caster as it is well known to those skilled inthe art. A simple ray-caster casts rays from a certain point of viewthrough a 3D-surface and renders the first surface the ray(s) meet. Moresophisticated ray-casters can take material into consideration and forexample even render a view as if it has passed through glass. However,for this embodiment of the invention a very simple ray-caster issufficient. Other methods of rendering 3D-images may also be employed.In case of this embodiment the 3D-surface is the multi-modalthree-dimensional map.

The point of view is a raw location. The raw location can be obtained byany known locating means, for example GPS, magnetic sensors or aninertial navigation system that uses motion sensors like accelerometersand gyroscopes. The raw location estimation can also be improved byconsidering part or all of the previous known precise locations in themap. This information can be used, together with geometrical constraints(like walls) of the map, to reduce uncertainty in current raw locationestimated from the sensor. For example, a person cannot stand wherethere are buildings or trees.

The artificial view and the real view are then being compared. If theyare essentially the same, the point of view of the artificial view isconsidered the traffic participant's location. If they are not the same,further artificial views are compared to the real view until a locationhas been determined.

One problem that can impede the aforementioned method to determine alocation is that the view of the camera that takes the picture that isused to create the real view is obstructed. These obstructions can beany objects that are between the surfaces and objects of the multi-modalthree-dimensional map and the camera. Usual causes for such anobstruction are dynamic objects that can be part of the scenery but arenot captured by the multi-modal three-dimensional map, such as cars 24,pedestrians 25 (see FIG. 3 for both), non-permanent art installations oradvertisements such as A-frames.

Even if the threshold to consider the real view and the artificial viewis set very coarse, a high number of dynamic objects can lead to falsenegatives when comparing views. To avoid this problem, when processingthe digital picture to create the real view a sub-step “i. Removing ofdynamic objects from the picture” is carried out. One possible means toremove dynamic objects is by identifying them with an artificialintelligence, for example a convolutional neural network that is trainedto identify pedestrians, cars, bicycles and the like in pictures.

The described method of determining a location can be advantageouslyimplemented independent of the invention. It can be used in combinationwith the aforementioned other characteristics of the invention or on itsown.

Once the precise location has been determined the relation between auser and its surroundings is known and an augmented reality thatprecisely fits reality can be created.

The user in this case does not need to be known in the art. It sufficesif he is simply capable of operating the device on which the method iscarried out, e.g. using a smartphone.

According to a preferred embodiment of the invention the beacons fromstep D are perceptible within said augmented reality.

For example, the perceptible beacons can be superimposed into the fieldof vision of a pair of smart glasses, within the picture that was takento determine the location or within a camera view of the aforementioneddevice.

However, these means are of little to no help to visually impaired orblind users. Therefore, according to a further preferred embodiment ofthe invention the method is characterized in that the augmented realityis acoustic and that the beacons are audible at their respectivelocations. This can be realized for example via headphones that simulatenoises at certain locations. In a simpler exemplary embodiment, thedevice itself generates a sound that gets louder when pointed towardsthe nearest beacon and lower when pointed away. When a beacon/waypointhas been reached a sound can be played to indicate that the sound forthe next beacon is now played. Using a stereo technique, the sound canalso be heard louder in the left speaker when the waypoint is at theleft of the user and louder in the right speaker when the waypoint is atthe right of the user. In a preferred embodiment, binaural virtual audio(also known as spatial audio or 3D audio) can be used. In this case, thedevice processes an audio source to simulate to the user that the soundis coming from the actual spatial location of the waypoint. As thistechnique simulates the way humans perceive direction and distance of asound source, it is ideal to guide users through the path since it is atrue means to implement virtual audio sources in space (audio augmentedreality).

It is also preferred that only one beacon is active at a time.Preferably only the closest beacon along the path is active and it isswitched to the next when the user reaches the respective waypoint.

The described method to guide people can be implemented advantageouslyindependent of the invention and used for guiding systems that are basedon other methods, for example for guiding people within a building. Apossible method to locate the user relative to his surrounding couldthen for example be the use of RFID transponders within the building anda corresponding RFID chip on the user.

The augmented reality can of course contain further indicators, forexample when reaching or crossing transitioning spaces or what type oftransitioning space there is (zebra crossing, traffic light, stairs,ramp, lift, escalators and so on . . . ).

In general, every element that is noted in the multi-modalthree-dimensional map can be part of the augmented reality.

A traffic participant according to the invention can not only be apedestrian but also a bicycle rider, a drone, a car or the like.

-   -   1 urban scene 14 street sign    -   2 pavement 15 path    -   3 building 16 pavement (not    -   4 building connected)    -   5 flower bed 17 waypoint    -   6 lamp posts 18 waypoint    -   7 cars 19 wall    -   8 route 20 edge/corner    -   9 path 21 tree    -   10 street 22 border    -   11 zebra crossing 23 waypoints    -   12 bollard 24 cars    -   13 trash bin 25 pedestrians

1. Computer implemented method for guiding traffic participants betweenat least two places wherein the method contains the following steps: A.providing a multi-modal three-dimensional map, B. calculating a routebased on the multi-modal three-dimensional map connecting the at leasttwo places over at least one intermediate waypoint, C. determiningprecise location of the traffic participant, and D. setting beaconsalong the path at the waypoints.
 2. The method according claim 1,wherein the multi-modal three-dimensional map is created from at leasttwo sources.
 3. The method according to claim 2, wherein the sources instep A contain at least one or more of the following: two-dimensionalmap data, satellite images, conventional three-dimensional information,topographic maps, (municipal) street tree inventories, hydrants plans,street constructions plans, surface texture for three-dimensional modelsand aerial views.
 4. The method according to claim 1, wherein within themulti-modal three-dimensional map at least one walkable space isdefined.
 5. The method according to claim 4, wherein step B is carriedout based on the walkable space within the multi-modal three-dimensionalmap.
 6. The method according to claim 4, wherein two walkableneighboring spaces are connected via at least one waypoint.
 7. Themethod according to claim 4, wherein during step A within themulti-modal three-dimensional map at least one transitioning space isdefined and that a transitioning space bridges a gap between at leasttwo walkable spaces that are not bordering each other.
 8. The methodaccording to claim 1, wherein at least one obstacle is identified andmarked in the multi-modal three-dimensional map and that at least onewaypoint is set to circumvent said obstacle.
 9. The method according toclaim 1, wherein step C is carried out with the following sub-steps: a.acquiring a real view by processing at least one digital picture of thesurrounding, b. generating at least one possible artificial view basedon a raw location, the raw location providing a scene that is part ofthe multi-modal three-dimensional map and that is being depicted toprovide the artificial view, c. comparing the artificial view and thereal view, d. if the artificial view and the real view are essentiallythe same, providing location as the point of view the artificial viewwas generated from, and completing the subprocess, e. if the artificialview and the real view are not essentially the same repeating thesubprocess with at least one different artificial view.
 10. The methodaccording to claim 9, wherein sub-step C.a. contains the sub-step i.removing of dynamic objects from the digital picture.
 11. The methodaccording to claim 10, wherein dynamic objects are one or more from thelist containing: cars, other pedestrians, animals, movable trash bins,A-signs, mobile advertising installations, bicycles and the like. 12.The method according to claim 1, wherein with the precise location ofstep C and the multi-modal three-dimensional map from step A anaugmented reality around a user is created and that the beacons fromstep D are perceptible in said augmented reality.
 13. The methodaccording to claim 12, wherein the augmented reality is acoustic andthat the beacons are audible at their respective location.
 14. Themethod according to claim 1, wherein only one or two beacons are activesimultaneously and that the active beacon or beacons are the mostproximate in direction of the path.
 15. System for guiding trafficparticipants, wherein a device carries out the method according toclaim
 1. 16. A non-transitory computer-readable medium on which isstored a series of instructions that, when executed by a computer, causethe computer to perform the method of claim
 1. 17. The method of claim1, wherein the determining step uses the multi-modal three-dimensionalmap.
 18. The method of claim 2, wherein the multi-modalthree-dimensional map is created from at least three sources.
 19. Themethod according to claim 2, wherein within the multi-modalthree-dimensional map at least one walkable space is defined.
 20. Themethod according to claim 3, wherein within the multi-modalthree-dimensional map at least one walkable space is defined.